In the recent years, by making use of the speed-up operation of an image sensor such as CMOS, digital cameras with various functions have been developed. For example, there is one capable of continuously storing five images or more per second or one capable of high-speed video shooting at 1000 fps or more. It is known that such digital cameras include an image processing unit to synthesize continuously shot images for the purpose of removing from images blurs caused by camera shakes or noise due to long-time exposure. By image synthesis, it is able to average random noise and reduce noise due to camera shakes or shooting in the dark.
To synthesize continuously shot images, it is necessary to decide one of the images as a reference image and align positions of the other images (comparative images) with that of the reference image. For images of a completely still subject, this position alignment is done by making sizes of the images equal to each other. However, for images including a moving subject, it is needed to determine a direction in which the subject is moving and then align the positions according to the motion of the subject. Image synthesis without subject's motion taken into account results in increasing blurs and noise in the image. Japanese Patent No. 3935500 (Reference 1) discloses an image processing unit which is configured to determine motion of a subject first and synthesize images while aligning positions of the images based on data on the motion, for example.
Further, to prevent an image from being crashed (ghosts or else) by image synthesis of a moving subject, there is a known image processing method of dividing each of a reference image and a comparative image into very small image blocks and determining a synthesis rate according to a difference in RGB output values of each image block. For example, Japanese Patent Application Publication No. 2009-164857 (Reference 2) discloses an image processing method to determine for each image block whether or not images should be synthesized, synthesize image blocks when a difference value in RGB output values (mean RGB values) of the image blocks is below a predetermined threshold and not to synthesize them when the difference value exceeds the predetermined threshold.
There is a problem with determining an image synthesis using a predetermined threshold as in the image processing method disclosed in Reference 2. Images are captured with an image sensor in which random noise occurs in proportion to an exposure amount so that noise in a light portion of a subject image is larger than that in a dark portion and an absolute value of a difference value in images is large. Accordingly, to correct noise in a light portion by image synthesis, a threshold for determining whether or not to execute image synthesis need be set to a large value, which realizes accurate correction of motion of a subject in the light portion of image. However, the large threshold set for the light portion is too large for the dark portion of a subject image, making it impossible to detect motion of a subject in the dark portion and leading to erroneous determination on the image synthesis. Based on the erroneous determination, images of a moving subject are synthesized and a crash (ghost or the like) occurs in the synthesized image.
FIGS. 14A to 14C and 15 show results of image synthesis using different thresholds. An image in FIG. 14A is a reference image, and although not shown, three continuously captured comparative images are used for the image synthesis. FIG. 14B shows a result of image synthesis when a threshold for determining image synthesis is set for a dark portion of an image, for example. In FIG. 14B a subject X is a moving subject and image synthesis is correctly determined so that profile of the image is clear and includes no crash such as ghosts.
Meanwhile, FIG. 14C shows a result of image synthesis when a threshold for determining image synthesis is set for a light portion of an image, for example. In FIG. 14C a moving subject X is dark and a difference value in the reference image and the comparative image is small and unlikely to exceed a large threshold set for the light portion. Because of this, the images are erroneously determined to be suitable for image synthesis, resulting in generation of an image with profiles R along the motion of the subject X or a ghost Z.
Further, when the threshold is set for a dark portion of an image, noise Y conspicuously occurs only in a light portion of the image, as shown in FIG. 15, for example. That is, a locally bright area such as headlights of a car or streetlights is determined to be unsuitable for image synthesis while the surrounding image blocks are determined to be suitable. Accordingly, the image blocks corresponding to the bright area are not synthesized and the averaging of noise is not achieved by the image synthesis so that noise is conspicuous in the locally bright area.
As described above, using a fixed threshold, suitability of images for image synthesis cannot be accurately determined and as a result, the image synthesis is not properly done. This induces occurrence of ghosts or distinctive noise in images and causes image crash.